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Cellular  Nonlinear  Networks  (CNNs)  are  large  arrays  of  nonlinear  circuits  coupled  to  their  im¬ 
mediate  neighbors.  During  this  funding  period  we  have  made  many  advances  in  understanding  the 
pattern-forming  dynamics  of  such  circuits  and  their  relationship  to  problems  in  physics  and  biol- 
ogy,  we  have  explicated  the  image  processing  capabilities  of  such  CNNs,  including  spatial  filtering 
and  multiscale  analysis,  and  finally  we  have  obtained  rigorous  mathematical  results  concerning  the 
dynamic  behavior  of  simple  CNN  arrays. 

Large  arrays  of  complex  cells  have  elsewhere  been  shown  to  demonstrate  interesting  pattern 
forming  behaviors —  such  as  the  react  ion- diffusion  systems  of  Turing,  the  propagation  of  autowaves, 
and  the  Ising  spin  system.  We  have  shown  that  the  simple  first-order  CNN  is  capable  of  exhibiting 
the  essential  features  found  in  these  systems.  And,  due  to  the  continuous-time  nonlinear  dynamics 
and  general  neighborhood  weights  the  patterns  formed  by  the  CNN  proved  to  be  a  study  in  their 
own  right. 

The  linear  spatial  convolution  operation  is  essential  in  all  manner  of  nonlinear  and  linear  image 
processing  algorithms.  Under  this  funding,  we  demonstrated  an  approach  by  which  any  arbitrary 
FIR  filter  could  be  implemented  in  a  robust  and  straightforward  manner  via  a  CNN  Universal 
Machine  (CNNUM)  algorithm  -  by  using  only  a  standard  3x3  B-template.  In  addition,  a  general 
canonical  form  was  proposed  for  such  CNN  linear  convolutions,  which  also  takes  into  account  the 
use  of  an  A-template  and  previous  approaches  to  convolution. 

Finally,  we  initiated  an  investigation  into  a  rigourous  analysis  of  the  CNN  dynamics  as  a  function 
of  the  template  parameters.  The  first  theorems  have  been  obtained  for  the  two-cell  CNN  case.  We 

expect  generalizations  of  such  results  to  verify  the  proper  behavior  of  many  CNN  image  processing  ■  '  . 
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Pattern  Formation  Properties  of  Cellular  Neural  Networks 

Pattern  formation  in  the  CNN  is  a  result  of  the  mechanism  of  symmetry  breaking  around  an 
unstable  equilibrium.  The  symmetry  can  be  broken  by  either  small  random  noise,  a  ‘seed’  or  site 
defect,  or  a  systematic  disturbance.  The  parameters  which  affect  the  type  of  pattern  formed  are 
the  bias  and  interconnection  weights. 

Random  Perturbation 

When  the  symmetry  breaking  is  introduced  as  a  small  random  perturbation,  our  research  has  led 
us  to  identify  the  following  three  epochs  of  behavior  in  the  time  formation  of  patterns. 

•  Linear  System  leading  to  noise  shaping. 

•  Separation  of  Spatial  Frequencies  and  Phases  leading  to  meta-stability. 

•  Boundary  Negotiation  leading  to  stability. 

Various  observations  and  theoretical  results  have  been  made  for  each  of  these  epochs,  which  are 
summarized  below: 

Linear  System 

If  the  system  is  halted  at  the  moment  when  the  first  cell  of  the  array  enters  the  saturation  region, 
the  linear  system  theory  will  be  exact.  The  states  at  this  time  wiU  be  the  result  of  a  linear  filtering 
operation  which  can  be  found  in  terms  of  the  template  weights.  Such  a  filtering  operation  can  be 
understood  to  enhance  certain  spatial  frequencies  while  suppressing  others. 

Motif  Separation 

In  regions  where  a  significant  number  of  cells  have  reached  saturation,  the  linear  analysis  does  not 
even  hold  approximately.  At  this  point,  arbitrary  combinations  of  the  unstable  modes  can  no  longer 
be  maintained.  The  states  begin  to  separate  themselves  into  regions  with  motifs  that  are  at  least 
locally  stable.  These  regions  grow  until  their  boundaries  meet  with  other  regions  of  different  motif. 
At  this  point  the  system  is  considered  to  be  meta-stable  in  the  sense  that  most  cells  are  unchanging 
in  time  (the  interior  cells  of  the  regions)  and  only  the  boundaries  between  regions  are  moving. 

It  is  difficult  to  determine  what  the  possible  meta-stable  motifs  are.  We  have  shown  that  if  any 
combination  of  equally  unstable  modes  has  a  purely  binary  representation,  then  in  the  saturation 
region,  that  motif  is  stable,  at  least  in  the  sense  that  a  big  enough  patch  of  it  will  not  change  unless 
outside  influences  force  it  to.  It  is  not  certain  that  a  finite  array  containing  this  motif  would  be 
stable,  however. 

For  a  3  X  3  template  of  weights  (i.e.  immediate  neighbors)  we  have  fully  characterized  the  stable 
motifs.  In  addition,  simple  inequalities  allow  the  determination  of  the  dominant  stable  motif,  which 
will  generally  arise  from  random  initial  conditions. 

Boundary  Negotiation 

Even  though  the  array  has  separated  into  regions  that  are  locally  stable,  the  boundary  between 
them  may  prove  to  be  unstable.  Thus  begins  a  long  process  where  most  of  the  cells  in  the  array  are 
not  changing  in  time,  but  slowly  the  boundaries  are  negotiated.  In  some  cases  there  may  not  be 
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any  acceptable  boundary,  and  eventually  one  motif  must  cover  the  whole  array.  The  whole  process 
may  take  many  order  of  magnitudes  longer  than  the  first  two  steps. 

Bias 

We  have  studied  some  of  the  effects  of  using  the  bias  term  in  the  dynamics.  The  bias  changes  the 
allowed  meta-stable  motifs.  For  instance,  we  have  used  this  fact  to  demonstrate  the  formation  of 
‘stripes’  of  a  tiger  when  no  bias  is  used  and  the  ‘spots’  of  a  leopard  when  bias  is  added.  Varying 
the  bias  in  time  can  simulate  some  of  the  effects  of  striping  patterns  in  the  angelfish.  Finally,  we 
have  begun  investigations  into  the  use  of  space- varying  bias  for  image  processing  applications  such 
as  fingerprint  enhancement. 

Site  Defects 

Finally,  we  have  investigated  the  mechanism  involved  in  the  production  of  spiral  and  target  patterns 
in  the  CNN  by  using  ‘seeds’  to  break  the  unstable  equilibrium.  When  one  site  of  the  array  is 
disturbed  from  equilibrium,  a  pattern  begins  propagating  outward.  This  will  usually  take  the  form 
of  a  circular  blob  containing  the  dominant  motif.  However,  when  more  than  one  seed  is  used  in  the 
same  array,  these  motifs  must  join  up  in  a  consistent  way,  which  can  cause  the  formation  of  spirals. 
The  patterns  formed  are  reminiscient  of  the  aggregate  growths  of  bacteria  colonies,  for  instance. 
Furthermore,  we  have  discoved  template  values  which  exhibit  dendritic  growth  from  a  site  defect, 
a  previously  unknown  phenomena  in  the  first-order  CNN  ceU  array. 

Linear  Spatial  Convolution 

As  linear  filtering  is  the  workhorse  of  image  processing  algorithms,  the  question  of  whether  the 
CNNUM  can  implement  any  convolution  and  how  to  do  so  is  important.  We  have  shown  how  these 
methods  can  be  applied  to  multiscale  image  processing. 

It  has  long  been  understood  that  the  B-template  of  the  simple  CNN  performs  a  correlation 
(reflected  convolution)  with  the  input  image.  However,  due  to  implementation  concerns  the  B- 
template,  and  therefore  the  convolution  kernel,  is  restricted  to  be  small  in  size  -  typically  3x3.  By 
also  using  the  A-template,  or  by  cascading  B-template  operations,  large  kernel  convolutions  can 
be  performed  but  the  coefficients  of  the  impulse  response  cannot  be  arbitrarily  specified.  However, 
the  need  to  exactly  specify  a  large  convolution  mask  arises  in  many  situations,  such  as  template 
matching  for  object  detection,  dilation  or  erosion  by  large  structuring  elements,  and  interpolation. 

Canonical  Form  for  General  FIR  Filtering 

We  have  developed  a  canonical  form  for  the  possible  spatial  linear  filtering  operations  which  can  be 
performed  on  the  CNN  Universal  Machine,  when  running  a  finite  algorithm  using  single-transient 
CNN  spatial  convolution  and  image  addition.  Convolution  by  arbitrary  kernels  of  any  finite  size  can 
be  shown  to  be  implementable  by  algorithms  of  this  form  using  only  3x3  templates.  Equivalently 
it  can  be  considered  a  general  form  capable  of  implementing  arbitrarily  large  B-template  effects  by 
using  only  3x3  connectivity. 

Previous  methods  for  performing  arbitrary  convolution,  which  involved  either  summations  of 
multiple  applications  of  small  B-templates  or  partitioning  and  shifting  can  be  unified  under  this 
form.  When  implementing  a  desired  convolution  with  this  form  the  choice  of  templates  is  under¬ 
determined  and  either  of  these  previous  methods  can  be  used  to  demonstrate  completeness.  We 


have  produced  a  CNNUM  algorithm  for  implementing  the  partition-shift  approach  with  special 
concern  given  to  practical  issues  of  addition  and  accuracy.  It  was  presented  both  as  a  constructive 
proof  of  the  general  convolution  capabilities  of  the  CNNUM  and  as  a  practical  approach  in  some 
cases.  The  unified  form  includes  a  broader  class  of  implementations  than,  for  instance,  those  that 
use  the  A-template  for  HR  filtering,  by  combining  multiple  single-transient  filtering  operations 
through  image  addition  and  cascaded  convolution  (i.e.  serial-parallel  forms). 

Multiscale  Analysis 

As  an  interesting  application  of  linear  filtering  on  the  CNN  Universal  Machine,  our  work  has 
also  included  the  investigation  of  some  methods  for  generating  multi-scale  representations  of  gray¬ 
scale  imagery  on  the  CNNUM.  The  method  uses  time-indexed  CNN  spatial  filtering  operations 
(analogous  to  ‘stopped  diflFusion’),  which  are  then  combined  under  the  canonical  form  by  image 
subtractions  and  cascaded  convolutions  to  produce  the  desired  filtering  operation. 

Size  and  relative  size  of  objects  are  among  the  basic  variables  in  image  analysis,  it  is  essential 
therefore  for  any  image  processing  system  to  be  able  to  extract  scale  related  information.  A 
powerful  way  to  obtain  scale-space  information  is  through  generating  and  analyzing  multi-scale 
representations  of  the  input  data.  Features  at  or  below  a  given  scale  are  normally  extracted  by 
convolution  operators  where  the  relative  size  of  the  image  and  the  convolution  kernel  determines 
whether  the  feature  will  be  extracted  or  suppressed. 

The  CNN  Universal  Machine,  can  be  used  for  multi-scale  processing  by  making  use  of  the 
dynamics  of  CNN  analog  computation  and  combining  the  results.  By  using  the  CNN  to  approximate 
the  heat  equation,  the  scale  parameter  of  the  linear  diflfusion  scale-space  is  translated  into  the 
running  time  of  a  transient  on  the  CNN.  Scale-space  filters  of  theoretically  arbitrary  size  can  be 
implemented  by  this  method.  These  filtered  images  can  then  be  combined  in  a  CNNUM  context 
to  produce  images  containing  only  selected  feature  scales.  Such  a  solution  allows  for  arbitrary 
quantization  of  the  scale  parameter  as  opposed  to  the  coarse  1:2  ratio  resulting  from  subsampling 
done  by  discarding  every  other  sample. 

The  investigated  approach  provides  for  easy  generation  of  various  multi-scale  representations.  In 
particular,  it  has  been  shown  how  the  CNNUM  can  be  used  to  approximate  Gaussian  and  Laplacian 
multi-scale  decompositions,  as  well  as  approximating  the  Continuous  Wavelet  Transform. 


Mathematical  Study  of  CNN  Dynamics 

During  this  period,  we  have  also  initiated  a  research  effort  to  provide  a  rigorous  mathematical  basis 
for  the  CNN  paradigm,  especially  for  image  processing  applications.  The  questions  which  we  would 
like  to  eventually  answer  include: 

1.  When  should  we  expect  a  CNN  to  have  stable  dynamics,  i.e.  when  will  the  system  settle 
to  a  steady  state?  This  depends  on  the  templates,  the  number  of  nodes  in  the  system,  and 
the  value  of  the  bias.  Previously,  we  have  obtained  some  results  in  this  area  (e.g.  symmetric 
templates,  cooperative  templates),  but  some  of  the  basic  cases  are  stiU  not  fully  understood. 

2.  How  does  the  qualitative  behavior  of  the  system  change  as  the  parameters  are  varied?  For 
instance,  what  happens  to  an  equilibrium  as  it  is  smoothly  pushed  out  of  the  central  linear 
region  of  state  space?  We  believe  that  ideas  from  the  field  of  bifurcation  theory  will  be  useful 
in  answering  these  types  of  questions. 


3.  Is  there  a  systematic  way  to  determine  the  functionality  of  a  CNN  from  the  system  equations 
or  template  values?  This  is  a  very  fundamental  and  important  question,  not  only  to  prove 
that  a  particular  template  has  the  desired  behavior,  but  because  even  a  partial  answer  can 
be  very  useful  in  designing  templates  for  various  applications.  We  envision  the  development 
of  a  ‘template  calculus’  which  allows  us  to  design  CNNs  with  complex  functionalities  from 
combinations  of  simple  templates. 


In  our  attempt  to  address  such  questions,  we  have  obtained  the  following  initial  results. 
The  two  node  cellular  neural  network  (CNN)  is  the  system  described  by  the  equations: 

i  -  -a:  +  ba{x)  +  ca{y)  +  ix 

y  -y  +  ba{y)  +  aa{x)  +  iy 


where  x  and  y  are  the  activations  of  the  cells,  a,  6,  and  c  are  parameters  (often  referred  to  as  the 
“A  template”),  ix  and  iy  are  the  node  inputs,  and  a{s)  is  the  function 


a{s)  = 


-1 

s 

1 


if  5  <  —1 
if-1  <  s  <  1 
if  1  <  s. 


Thus  a  particular  CNN  is  one  where  a,  b,  and  c  are  chosen  and  fixed,  and  x  and  y  are  allowed  to 
vary  with  time,  subject  to  the  differential  equations  above.  In  general,  the  inputs,  4  and  iy^  may 
be  either  constant  or  aUowed  to  vary  with  time.  For  the  theorems  stated  below,  ix  =  0  and  iy  =  0. 

There  is  much  experimental  data  about  the  behavior  of  CNN’s,  for  two  nodes  or  many  nodes,  but 
the  mathematical  model  of  them  is  not  weU  understood  for  many  possible  sets  of  parameters,  either 
with  or  without  external  inputs  (ix  and  iy).  There  are  some  previous  results  which  characterize 
the  behavior  of  CNN’s  for  various  parameter  values  (for  instance  when  a  —  c).  We  have  extended 
this  set  of  results  as  follows: 


Theorem  1  For  a  two  node  CNN  with  no  inputs,  if  b  <  I  then  the  system  is  convergent,  with  a 
global  attractor  at  (0,0). 

Theorem  2  For  a  two  node  CNN  with  no  inputs,  in  the  parameter  range  given  by  b  >  I,  ac  <  0, 
b  <  max{c-{- 1,  —c  +  1},  and  b  <  max{a+  1,  — a  +  1),  there  are  at  most  two  cycles  which  lie  entirely 
outside  the  box  |a:|  <  lj\y\  <  L 

The  first  result  is  an  application  of  the  divergence  theorem.  Unfortunately,  this  application  is 
only  valid  in  two  dimensions.  However,  the  knowledge  that  these  systems  are  convergent  may  help 
to  conclude  that  higher  dimensional  CNN’s  are  also  convergent  when  6  <  1.  The  similarity  of  the 
behavior  for  all  of  these  parameter  values  has  led  us  to  look  for  a  Lyapunov  function  which  will  be 
valid  whenever  6  <  1.  This  line  of  inquiry  looks  promising.  If  such  a  Lyapunov  function  is  found, 
it  is  likely  that  it  wiU  be  generalizable  to  higher  dimensional  systems. 

For  the  second  result,  the  piecewise  linear  structure  of  the  equations  was  exploited.  Since  the 
(x,  y)  plane  may  be  divided  into  regions  where  the  differential  equations  are  linear,  solutions  may  be 
written  down  which  are  valid  as  long  as  a  trajectory  stays  within  one  of  these  regions.  A  complete 
solution  may  then  (in  theory)  be  constructed  through  any  point  {xo,yo)^ 

Unfortunately,  there  is  a  serious  barrier  to  constructing  the  solution  through  a  particular  initial 
condition  as  a  function  of  the  parameters.  To  construct  a  solution,  one  starts  with  the  knowledge 
of  the  general  solutions  to  the  differential  equations  that  are  valid  in  each  region  of  the  phase  space 


(as  mentioned  above,  the  equations  are  linear  in  each  region).  Then  one  ‘‘steps”  between  particular 
solutions  of  these  equations  to  construct  a  complete  solution.  This  process  involves  finding  the 
coordinates  of  the  point  where  a  particular  solution  through  {xq^Vo)  intersects  the  boundary  of  a 
region.  For  most  choices  of  (a^o^l/o)?  this  would  involve  finding  the  zeros  of  a  polynomial  of  degree 
\b-ll  and  so  this  procedure  cannot  be  carried  out  in  general. 

However,  for  some  initial  conditions,  the  equation  is  a  polynomial  of  degree  two  rather  than 
\b  —  1|,  and  can  be  found  in  closed  form.  This  makes  it  possible  to  find  the  Poincare  (first  return) 
map  for  the  ray  a:  =  1,  y  >  1,  but  only  for  points  whose  trajectories  remain  outside  the  center 
box.  If  one  or  more  cycles  exists  entirely  outside  the  center  box  (where  |3:|  <  l,\y\  <  1)  for  the 
parameter  values  described  in  Theorem  2,  then  they  wiU  be  fixed  points  of  this  map.  The  map  is 
a  composition  of  maps  of  the  form  (py  +  q)l{Ty  +  t)^  so  we  may  conclude  that  it  has  at  most  two 
fixed  points,  and  thus  at  most  two  periodic  orbits  (which  gives  Theorem  2). 

The  same  methods  which  give  Theorem  1  can  be  used  to  disallow  some  types  of  cycles  for 
other  parameter  values,  but  we  do  not  yet  fully  understand  the  possibilities  in  these  cases.  We  are 
currently  using  these  and  other  methods  to  characterize  behavior  in  the  remaining  cases  as  fully 
as  possible.  Other  work  in  progress  includes  calculating  the  boundaries  of  basins  of  attractions  for 
systems  with  more  than  two  nodes.  This  is  computationally  prohibitive  for  large  systems,  since  the 
calculation  involves  solving  3^  differential  equations,  but  knowledge  of  how  complicated  the  basin 
boundaries  may  be  for  smaller  systems  is  bkely  to  help  with  the  understanding  and  design  of  larger 
cellular  neural  networks. 
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